Papers with text-to-image retrieval
Fine-grained Image Captioning with CLIP Reward (2022.findings-naacl)
Copied to clipboard
| Challenge: | Modern image captioning models are usually trained with text similarity objectives . reference captions often describe only the most salient objects in images . |
| Approach: | They propose to use CLIP to calculate multi-modal similarity and use it as a reward function . they propose a simple finetuning strategy to improve grammar that does not require extra text annotation. |
| Outcome: | The proposed model generates more distinctive captions than the CIDEroptimized model on text-to-image retrieval and fineCapEval. |
Do Vision-Language Models Understand Compound Nouns? (2024.naacl-short)
Copied to clipboard
| Challenge: | Open-vocabulary vision-language models (CLIP) are emerging as a promising new paradigm for text-to-image retrieval. |
| Approach: | They propose a benchmark to evaluate the effectiveness of open-vocabulary vision-language models (CLIP) for text-to-image retrieval using contrastive loss. |
| Outcome: | The proposed framework improves CN understanding of CLIP by 8.25% on Compun. |
FEIDEGGER: A Multi-modal Corpus of Fashion Images and Descriptions in German (L18-1)
Copied to clipboard
| Challenge: | Recent years have seen a renewed interest in text-image multi-modality . paired text-picture datasets are often limited to English language text . |
| Approach: | They propose a multi-modal corpus that pairs images and textual descriptions of their content in German to enable study of these challenges. |
| Outcome: | The proposed dataset focuses on the domain of fashion items and their visual descriptions in German. |
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Contrastive language-image pre-training models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval. |
| Approach: | They propose a fine-tuning approach to enhance the representations of CLIP models for paraphrases by leveraging large language models. |
| Outcome: | The proposed model improves on baseline models across paraphrased retrieval, visual genome relation and attribution, and seven semantic textual similarity tasks. |
Learning the Visualness of Text Using Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual text evokes an image in a person’s mind, while non-visual text fails to do so. |
| Approach: | They propose a method to automatically detect visualness in text to enable text-to-image retrieval and generation models to augment text with relevant images. |
| Outcome: | The proposed method performs better than several baseline models and heuristics for the task. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to model multimodal data do not leverage cross-modal information . augmenting input text using cross-module attribute insertions results in poor performance . |
| Approach: | They propose a multimodal deep learning approach that adds visual attributes to inputs to enhance model robustness. |
| Outcome: | The proposed approach is modular, controllable, and task-agnostic. |
SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) relies on text-to-image retrieval using specialized bi-encoders . et al., 2022, 2024, 2021, 2023, 2026, 2030, 2040, 2050, 2060) document retrieval bridges human or artificial agents to the most relevant information, authors say . |
| Approach: | They propose a zero-shot visual document retrieval method that uses bi-encoders . they propose 63.4% nDCG@5 for visual document capture and a reusable semantic proxy . |
| Outcome: | The proposed method surpasses the strongest specialised multi-vector visual document encoder on the ViDoRe-v2 benchmark and scales similarly on MIRACL-VISION with broader multilingual coverage. |
Evaluating Perspectival Biases in Cross-Modal Retrieval (2026.findings-acl)
Copied to clipboard
Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant, Chompakorn Chaksangchaichot, Patawee Prakrankamanant, Witthawin Sripheanpol, Pak Lovichit, Sarana Nutanong, Ekapol Chuangsuwanich
| Challenge: | a recent study shows that multimodal retrieval systems are expected to operate in a semantic space, agnostic to the language or cultural origin of the query. |
| Approach: | They introduce a benchmark to quantify linguistic and cultural biases in multimodal retrieval systems . they propose a framework to decouple language from culture and decouples it from semantics . |
| Outcome: | The proposed benchmark systematically measures the effects of linguistic and cultural biases on retrieval performance. |